A General Distributed Dual Coordinate Optimization Framework for Regularized Loss Minimization

In modern large-scale machine learning applications, the training data are often partitioned and stored on multiple machines. It is customary to employ the “data parallelism” approach, where the aggregated training loss is minimized without moving data across machines. In this paper, we introduce a novel distributed dual formulation for regularized loss minimization problems that can directly handle data parallelism under the distributed computing environment. This formulation allows us to systematically derive dual coordinate optimization procedures, which we refer to as Distributed Alternating Dual Maximization (DADM). The method extends earlier studies described in [1, 7, 11, 20] and has a rigorous theoretical analysis. Based on the new formulation, we also develop an accelerated DADM algorithm by generalizing the acceleration technique from [16] to the distributed setting. Our empirical studies show that our novel approach significantly improves the previous state-of-the-art distributed dual coordinate optimization algorithms.